What You Need Are Active Speakers! Or Do You